A two-stage algorithm for the early detection of zero-quantized discrete cosine transform coefficients in High Efficiency Video Coding

For High Efficiency Video Coding (HEVC) using block-based prediction, the discrete cosine transform (DCT), and quantization, a large number of DCT coefficients in a transform block (TB) are commonly found to be quantized to zero. A two-dimensional transform in HEVC is usually implemented by first applying a butterfly-based one-dimensional (1D) DCT to each row of the residual block, followed by a 1D DCT to each column. Accordingly, we propose a two-stage method for the early detection of zero-quantized DCT coefficients in this paper. In the first stage, a distribution-based method, which uses the sum of absolute differences as the threshold criterion and compares it to threshold values obtained for all columns, is employed to detect zero-quantized columns in the pixel domain prior to conducting the actual DCT. After conducting the 1D row DCT, the second detection stage, which uses the intermediate matrix resulting from the row transform as the input, is applied in the transform domain only to those columns predicted to contain non-zero coefficients. As an orthogonal transform, the 1D row DCT has a tendency to pack a large fraction of the signal energy into a relatively few coefficients. Therefore, the second stage of our algorithm is more effective than methods that conduct detection in the pixel domain, particularly for TBs greater than 8× 8. Experimental results demonstrate that an HEVC encoder with our proposed two-stage algorithm can dramatically reduce unnecessary 1D row and column DCT operations compared with a standard encoder and therefore exhibits applicability to practical HEVC encoder implementation.